System and method for detection of anomalies in test and measurement results of a device under test (dut)

ABSTRACT

A test and measurement device has an interface, one or more connectors, each connector to allow the test and measurement device to connect to a test and measurement instrument, and one or more processors, the one or more processors configured to execute code to cause the one or more processors to: receive one or more user inputs through the interface identifying one or more tests to perform on a device under test (DUT); form a connection through one of the one or more connectors to the DUT to perform the one or more tests and receive test result data; apply one or more machine learning models to the test result data to identify potentially anomalous test results; and generate and present a representation of the test result data and the potentially anomalous test results. A method of analyzing test data includes receiving one or more user inputs through an interface identifying one or more test to perform on a device under test (DUT), forming a connection to at least one test and measurement instrument, directing the test and measurement instrument to perform one or more tests on the DUT and receive test result data, applying one or more machine learning models to the test result data to identify potentially anomalous test results, and generating and presenting a representation of the test result data and the potentially anomalous test results.

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure claims benefit of Indian Provisional Application No.202121043152, titled “SYSTEM AND METHOD FOR DETECTION OF ANOMALIES INTEST AND MEASUREMENT RESULTS OF A DEVICE UNDER TEST (DUT),” filed onSep. 23, 2021, the disclosure of which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to testing of electronicdevices and specifically to monitoring and analysis of electronic devicetesting data.

BACKGROUND

Flawless hardware designs of electronic components such as semiconductorchips often need multiple steps. One of the critical steps in thehardware design cycle involves robust validation, system, andverification test. The validation test cycle uncovers potentialanomalies critical for the release of the semiconductor chip.

FIG. 1 illustrates a typical semiconductor chip development cycle. Theseinclude a verification process that includes specification developmentand Resistor Transistor Logic (RTL) design, a validation step thatincludes stress testing and characterization, and a production step thatincludes manufacturing testing. The chip development cycle for System ona Chip (SoC) includes a SoC verification test, a SoC validation test,and a SoC production test. Further, the validation and compliance testcycle includes performance of a system test, a stress test, acharacterization test and a compliance test.

In the validation step, the characterization process involves testingthe design with voltage and frequency shmooing to find the idealoperating conditions. Designs with high-speed inputs and outputs (likePCIE, Ethernet, DDR, etc.) also go through characterization of IO portsby shmooing various electrical parameters to arrive at idealtransmission and error rates. This test involves large test cases andconsumes significant amount of the user's time. The post-analysis timeconsumed requires users to analyze the large datasets collected as partof the test, making the delay significant and increases the costs.

While executing compliance or characterization tests for any DeviceUnder Test (DUT) belonging to any given technology, test automationsoftware generally reports a set of measurements. Each of thesemeasurements could be compared against their specific limit values forobtaining individual measurement quality and all measurement resultsonce the execution is completed and later analyzed for gatheringspecific insights. This analysis takes up a lot of time and resources,increasing the costs mentioned in the above discussion.

By considering all measurement results instead of looking at individualresults and drawing siloed conclusion, the process can determine the DUTquality. In addition, with the above operation happening in anintegrated real time, it can significantly reduce the users/expert'seffort in debugging the problem.

Some problem solutions that achieve maximum automation of analysis oftest and measurement using Machine Learning (ML) and ArtificialIntelligence (AI). However, Test and Measurement (TNM) as a domainremains largely untouched with the latest developments and capabilitiesof ML probably due to the characteristics of the data and/or problemsassociated with TNM.

Therefore, a need exists for effective, automatic testing system andmethod based on ML/AI that provides smart data analysis of the test dataand a feedback near to real-time during the test execution. This cansignificantly reduce the users/expert's effort in debugging the problem,speed up the process, and make the process more efficient.

BRIEF DESCRIPTION OF ACCOMPANYING DRAWINGS

FIG. 1 shows a conventional semiconductor chip development cycle inaccordance with the prior art.

FIG. 2 shows an embodiment of a test and measurement system.

FIG. 3 shows an architecture of a system for detection of anomalies intest and measurement results of a Device Under Test (DUT) in avalidation setup.

FIG. 4 shows an embodiment of an ML/AI based data analysis operationbased on a level of intensity for detection of anomaly/outliers of testand measurement results.

FIG. 5 shows a representation of moving window of data sets for onlinetraining and prediction for identifying outliers.

FIG. 6 shows a graph indicating the normal and anomalous behavior whileperforming PCIE Gen2/3/4 signal test execution.

DETAILED DESCRIPTION

The various embodiments of the present disclosure provide a machinelearning—artificial intelligence (ML/AI) based system and method fordetection of anomalies in test and measurement results of a DUT. Thepresent disclosure provides a ML/AI method and system for detection ofanomalies in test and measurement results of a DUT in real time bytaking into consideration consolidated reported measurements with thetest.

The discussion here uses the terms artificial intelligence (AI) andmachine learning (ML) mostly interchangeable. Machine learning generallymakes up subset of AI. ML involves a model-based system in which themodel may undergo training, often referred to as supervised learning, to“teach” the model how to recognize patterns and make predictions basedupon data received. Unsupervised model-based systems do not typicallyhave a training process. For example, the conditions surrounding aparticular data set gathered during operation determines how the modeldetermines its predictions. This discussion may refer to the machinelearning module as a machine learning network, as it will generally takethe form of a deep learning network, where the term “network” refers toa network of “nodes” or “neurons” in the machine learning module.

In the following description, for purpose of explanation, specificdetails are set forth to provide an understanding of the presentdisclosure. It will be apparent, however, to one skilled in the art thatthe present disclosure may be practiced without these details. Oneskilled in the art will recognize that embodiments of the presentdisclosure, some of which are described below, may be incorporated intoseveral systems. However, the systems and methods are not limited to thespecific embodiments described herein. Further, structures and devicesshown in the figures are illustrative of exemplary embodiments of thepresent disclosure and are meant to avoid obscuring of the presentdisclosure.

Furthermore, connections between components and/or modules within thefigures are not intended to be limited to direct connections. Rather,these components and modules may be modified, re-formatted or otherwisechanged by intermediary components and modules. References in thepresent disclosure to “embodiment” mean that a particular feature,structure, characteristic, or function described in connection with theembodiment is included in at least one embodiment of the disclosure. Theappearances of the phrase “in an embodiment” in various places in thespecification are not necessarily all referring to the same embodiment.

Generally, the system platform will reside on a test and measurementcomputing device that may connect to multiple test and measurementinstruments as the various testing needs arise. As technology continuesto evolve and more computing power can fit into smaller and lessexpensive packages, the embodiments may encompass a single device orhousing that contains all of the necessary hardware and components toact as both the test and measurement computing device and the test andmeasurement instrument connected directly to the DUT. No limitation toany particular architecture is intended and none should be implied.

The embodiment of FIG. 2 shows the two devices as separate devices, withthe understanding that elements of each may be combined and all of thefunctionality and capability may reside in one device. The test andmeasurement device 10 may comprise a computer, server or other computingdevice. The toolkit device 10 may include one or more processors 12 thatcommunicate with a test and measurement instrument 20 through a port 14.This port may comprise a wireless or wired/cabled port combined withdrivers for multiple test and measurement instruments that allow thetoolkit device to set up and run test on a device under test (DUT),including connecting directly to the DUT. The port 14 may include portsto communicate with other devices, remote storage, etc. Alternatively,another data port 13 may provide that access.

The toolkit device will also include one or more data repositories,represented by the memory 16. The user interacts with the test andmeasurement device through a user interface 18 and a display 19. Thisallows the users to make selections for tests and displayrepresentations of the data analysis, such as by visualizationsperformed by the machine learning system or data readouts through theAPI. The test and measurement instrument 20 has similar components tothe toolkit device, which may combination of the two devices in someembodiments, as mentioned previously.

The one or more processors 12 may be configured to execute code thatcauses the processors to implement the methods and system of theembodiments. The devices may distribute the processing tasks across bothdevices.

FIG. 3 shows an embodiment of a system architecture for detection ofanomalies in test and measurement results of a DUT. The system willtypically take the form of a tool, hereinafter referred to as “smartdata analysis tool” or toolkit. The architecture shown in FIG. 3 depictsthe integration of the smart data analysis tool in a validation setupinvolving the test and measurement results. As discussed above, this mayreside on a computing device separate from a test and measurementinstrument, or integrated into the instrument.

The smart data analysis tool comprises of an interface and an automationunit to perform one or more tests on a DUT. The interface enables a userto provide the required inputs and specify the test requirements. In oneembodiment, the interface contains a Tektronix® app, Tek automationSoftware Development Kit (SDK) 30. The interface interacts with theautomation unit 32 via an Open Application Program Interface (Open API)36 using one or more REST API calls. A REST API is an API that complieswith REepresentational State Transfer (REST) architectural style, usedto provide standards between computer systems on the web, making iteasier for them to communicate. The user can retrieve the anomaliesthrough a user interface such as an application 30 as mentioned above,or just through the Open API 36. The discussion here groups thesetogether using the term “interface.”

One or more third party applications 34 also interact with theautomation unit 32 via the Open API. The automation unit contains one ormore modules to perform the tests specified by the user, record testdata and perform analysis of the test data. In one embodiment, theautomation unit comprises of a Core Execution Engine 38, an analyticsengine 40, several services and repositories, and an Anomaly detectionmodel module 42. The analytics engine performs various analyses on thetest results data received from the DUT including various measurementsand comparison. The Anomaly detection module 42 will comprise a machinelearning network to apply machine learning to the data to provide thepredicted outliers analysis previously mentioned.

The various services include a report service, a provenance service, alog service, and an analysis service. As used here, the term “service”means software that performs automated tasks, responds to hardwareevents and requests from other software modules within the automationunit. In the embodiments here, these services include a report service,an analysis service, a provenance service, and a log service. The term“provenance” refers to information about the data including the testresults data and generally can be considered a form of metadata.

The Core Execution Engine 38 also has connections to variousrepositories that may also relate to the services discussed above. Theseincludes the configuration repository that stores test configurations.The data repository generally stores the test results data gathered fromthe test and measurement instrument. The provenance repository storesthe provenance information about the data and the log repository storesthe operation logs that may provide information about issues in thesystem operation. These repositories may reside in one memory asdemonstrated in FIG. 2 , such as partitions of a much larger memory, ormay represent separate memory devices for each repository.

The Anomaly detection model module is connected to the data repositoryfor performing analysis of the test data stored in the data repository.The Analytics engine performs “standard” analyses on analysis on thetest results data, in contrast with the Anomaly detection module, whichuses the machine learning network to perform analysis of the data andprovide the predictions of what data results are outliers.

The Core Execution Engine is further connected to an Instrument hub,which is connected to one or more instruments such as an oscilloscope44. In one embodiment, the instruments that are connected to theInstrument hub may include an oscilloscope, a bit error rate tester(BERT), and other testing instruments manufactured by Tektronix,Keithley or other manufacturers. Each of these instruments communicatewith the Instrument hub using some sort of communication protocol. Inone embodiment, the protocol is JavaScript Object Notation (JSON)commands over one or more TCP communication links or through a ProgramInstruction over a TCP, VXL11, GPIB and/or a serial communication link.The discussion here refers to the means of communicating and/oroperating the instruments as “connectors.” These may take the form of adriver that communicates across a wireless or wired/cabled connectionwith the test and measurement instrument, or they may comprise internalsoftware code that causes the device to perform the tests, if integratedinto the test and measurement device.

A typical test/analysis operation proceeds below with respect to thevalidation setup of FIG. 3 . A user desiring to perform smart dataanalysis for a Device Under Test (DUT) in a validation setup, turns ONthe smart data analysis functionality of the tool. The user usuallyaccomplished this by making selections on the user interface of the testand measurement device. The user then executes the tests scheduled inthe testing process. Execution of the tests will involve the test andmeasurement device connecting to the test and measurement instrument orinstruments connected to the DUT. As mentioned previously, this mayinvolve the test and measurement device connecting to a separate testand measurement instrument or initializing itself to start the test, ifthe toolkit is integrated into the instrument. When the test andmeasurement device obtains new results every newly acquired waveform,the result undergoes smart data analysis that identifies the potentialanomalies/outliers relative to its immediate neighbors. In oneembodiment of the present disclosure, the user can specify a requiredlevel of intensity for classifying anomaly/outliers of the test results,using anomaly detection models.

FIG. 4 shows a data analysis operation/method based on a level ofintensity for the detection of anomaly/outliers of the test results. Theanalysis uses anomaly detection models. The intensity levels become avariable that affects the analysis method to detect one or more outliersfrom the test results. As shown in FIG. 4 , the current embodiments usethree levels of intensity, but could use many more or fewer levels.

In an example to aid with understanding of the levels, the DUT has arequired range of a measurement it has to pass. A first level comprisesthe “conservative” level in which the device considers outliers as rareincidents. Accordingly, the conservative model categorizes a point asoutlier only if there is a very strong indication. In a “balanced”model, the model categorizes a point as anomalous provided there issufficient strength of such an indication. In the “aggressive” model,data is classified as extremely eager to categorize any point asanomalous with the slightest of indication. The user may influence theindications based upon ranges of measurements t given to the system thatmay cause some points to be a likely outlier. For example, using ajitter measurement as an example, the range could be 10-50 dB based upona particular design specification. This would result in a conservativeresult in identifying only those points laying outside or near thoselimits as outliers. Tightening the range, for example to 15-20 dB, orother levels may influence how the model operates to identify outliers.The selected model may be stored in the Model Store.

In order to monitor and flag the potential anomalies with DUT testresults in a consolidated manner, one embodiment combines all theavailable measurements. The set of measurements associated with a giventest serves as indicators towards the overall quality of the DUT and itscombination is imperative for the subsequent modelling. The measurementcombination may take the form of a normalized vectorization procedurewith dimension of the resultant vector being the total number ofmeasurements available for that test as an N-dimensional vector, where Nis the number of measurements. The number of data points may alsoinfluence the findings of data points being outliers, as the more datathe models have to operate upon, the more the outliers will benoticeable against the rest of the data.

In one embodiment, all the test results are stored in the datarepository. Then, based on the level of intensity specified by the userfor anomaly/outlier classification, the machine learning networkclassifies the stored data from the repository using the anomalydetection model into conservative, balanced and aggressive models basedupon the user's input in the test set up.

In addition to running the machine learning network, the machinelearning network will typically undergo training. In training the MLnetwork receives data sets and the “answers” that allow the network toconnect the input data to the resulting outputs. After training, thenetwork will have gained the ability to analyze the inputs to provide aprediction as to the nature of the output. In the machine learning here,it will predict whether a given data point comprises an outlier or not.

FIG. 4 illustrates a detailed representation of moving window basedonline training and prediction for identifying outliers. Theidentification of outliers is always relative to the immediate neighborsof test results just before the test of interest. It is important toperform identification of local outliers rather than globally becausevery often a sudden change in the behavior could become a norm in thelonger run. The training data sets identify the local outliers using awindow approach. The window approach uses a pre-defined window length ofprevious executed test results used for training. The data containedbetween lines 50 and 52 each make up one window, and those windowsoverlap. The approach enables adapting quickly to the changes on theinput at the same time hard limits on the amount of data for training,thereby giving fine-grained control for training performance. In oneembodiment, the window moves with a predefined resolution, typically one(1), meaning that the window represents one data point

In order to facilitate the online learning, the system needs to storethe incrementally updated models and retrieve them back at instant.Similarly, the collected data also needs to be backed up for a futureuse cases as a best practice. The device satisfies these requirements byregular synchronization of the learned model into a data repository andcurrent model stored in the model/configuration store where the archivedmodel is saved along with certain meta data. The vectorized test resultsused for training are also saved into the data repository as an inherentand silent dataset creation. The models undergo initial training,referred to here as pre-training using pre-trained data sets, and thenincremental training during model deployment as the system gathers moredata for data sets. This increases the accuracy of the models and trainsthem to cover new anomalies not in the pre-trained data sets

In one embodiment, an automation platform incorporating the smart dataanalysis tool was enabled for validation of PCIE Gen2/3/4 signal testexecution workflow. For every preset execution of signal test, anN-dimensional vector of measurement results was generated, and this wasused as inputs towards the Anomaly detection models. One should notethat an experiment involved PCIE for testing, but this solution appliesto all technologies.

FIG. 6 illustrates a graph indicating the normal and anomalous behaviorwhile performing the PCIE Gen2/3/4 signal test execution. Every datapoint of signal test result is given an anomaly score with higher scorebeing termed as outliers. The current prediction along with the previousprediction are visualized as a trend with solid dots for normal behaviorand hollow dots with potential 1% of anomalous behavior. The testresults data, meaning the anomalies, of the process may comprise a graphas shown in FIG. 6 , or may be a data readout highlighting theanomalies. These are referred to here as representations of the testresult data.

The embodiments of a ML-based system and method for automated testresult analysis for detection of anomalies in test and measurementresults of a DUT enable receiving better insights towards potentialanomalous behavior of the DUT. The method and system look at the DUTquality by taking all measurement results as indicator rather beingspecifically associated with one result. The embodiments include afeedback and course correction with near real-time latency for DUTdesign. Furthermore, the test and measurement device also provides theadvantages of integrated inherent insights, near real-time feedback andcourse correction, shorter turnaround time of the analysis cycle, andbetter-quality expectation of the DUT.

Estimating actual time saved in the deployment of such a device becomesdifficult until widespread use. However, typical validation analysisturnaround times have reached up to 20 days. The use of this systemshould reduce that time to a matter of one or two days.

Aspects of the disclosure may operate on a particularly createdhardware, on firmware, digital signal processors, or on a speciallyprogrammed general purpose computer including a processor operatingaccording to programmed instructions. The terms controller or processoras used herein are intended to include microprocessors, microcomputers,Application Specific Integrated Circuits (ASICs), and dedicated hardwarecontrollers. One or more aspects of the disclosure may be embodied incomputer-usable data and computer-executable instructions, such as inone or more program modules, executed by one or more computers(including monitoring modules), or other devices. Generally, programmodules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types when executed by a processor in a computer or otherdevice. The computer executable instructions may be stored on anon-transitory computer readable medium such as a hard disk, opticaldisk, removable storage media, solid state memory, Random Access Memory(RAM), etc. As will be appreciated by one of skill in the art, thefunctionality of the program modules may be combined or distributed asdesired in various aspects. In addition, the functionality may beembodied in whole or in part in firmware or hardware equivalents such asintegrated circuits, FPGA, and the like. Particular data structures maybe used to more effectively implement one or more aspects of thedisclosure, and such data structures are contemplated within the scopeof computer executable instructions and computer-usable data describedherein.

The disclosed aspects may be implemented, in some cases, in hardware,firmware, software, or any combination thereof. The disclosed aspectsmay also be implemented as instructions carried by or stored on one ormore or non-transitory computer-readable media, which may be read andexecuted by one or more processors. Such instructions may be referred toas a computer program product. Computer-readable media, as discussedherein, means any media that can be accessed by a computing device. Byway of example, and not limitation, computer-readable media may comprisecomputer storage media and communication media.

Computer storage media means any medium that can be used to storecomputer-readable information. By way of example, and not limitation,computer storage media may include RAM, ROM, Electrically ErasableProgrammable Read-Only Memory (EEPROM), flash memory or other memorytechnology, Compact Disc Read Only Memory (CD-ROM), Digital Video Disc(DVD), or other optical disk storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, and any othervolatile or nonvolatile, removable or non-removable media implemented inany technology. Computer storage media excludes signals per se andtransitory forms of signal transmission.

Communication media means any media that can be used for thecommunication of computer-readable information. By way of example, andnot limitation, communication media may include coaxial cables,fiber-optic cables, air, or any other media suitable for thecommunication of electrical, optical, Radio Frequency (RF), infrared,acoustic or other types of signals.

Additionally, this written description refers to particular features. Itis to be understood that the disclosure in this specification includesall possible combinations of those particular features. For example,where a particular feature is disclosed in the context of a particularaspect, that feature can also be used, to the extent possible, in thecontext of other aspects.

Also, when reference is made in this application to a method having twoor more defined steps or operations, the defined steps or operations canbe carried out in any order or simultaneously, unless the contextexcludes those possibilities.

EXAMPLES

Illustrative examples of the disclosed technologies are provided below.An embodiment of the technologies may include one or more, and anycombination of, the examples described below.

Example 1 is a test and measurement device, comprising: an interface;one or more connectors, each connector to allow the test and measurementdevice to connect to a test and measurement instrument; and one or moreprocessors, the one or more processors configured to execute code tocause the one or more processors to: receive one or more user inputsthrough the interface identifying one or more tests to perform on adevice under test (DUT); form a connection through one of the one ormore connectors to the DUT to perform the one or more tests and receivetest result data; apply one or more machine learning models to the testresult data to identify potentially anomalous test results; and generateand present a representation of the test result data and the potentiallyanomalous test results.

Example 2 is the test and measurement device of Example 1, furthercomprising one or more repositories to store information related to atleast one of the test result data, machine learning modelconfigurations, metadata, and system logs.

Example 3 is the test and measurement device of either of Examples 1 or2, wherein the one or more processors reside on a computing deviceseparate from the test and measurement instrument.

Example 4 is the test and measurement device of any of Examples 1through 3, wherein the test and measurement device and the test andmeasurement instrument are the same device.

Example 5 is the test and measurement device of any of Examples 1through 4, wherein the connectors comprise instrument drivers for one ormore test and measurement instruments.

Example 6 is the test and measurement device of Example 1, wherein thecode that causes the one or more processors to apply one or more machinelearning models comprises code that causes the one or more processorsto: generate an N-dimensional vector of the test result data for everytest performed; and assign an anomaly score to each data point in thetest result data.

Example 7 is the test and measurement device of any of Examples 1through 6, wherein the code the causes the one or more processors toreceive the one or more user inputs comprises code that causes the oneor more processors to receive an intensity level.

Example 8 is the test and measurement device of Example 7, wherein thecode that cause the one or more processors to apply one or more machinelearning models to the test result data to identify potentiallyanomalous test results causes the one or more processors to identifyoutliers depending upon the intensity level.

Example 9 is the test and measurement device of any of Examples 1through 8, wherein the one or more processors are further configured toexecute code that causes the one or more processors to train the one ormore machine learning models.

Example 10 is the test and measurement device of Example 9, wherein thecode that causes the one or more processors to train the one or moremachine learning models uses multiple windows of a pre-defined windowlength of previous executed test results.

Example 11 is the test and measurement device of Example 9, wherein tthe code that causes the one or more processors to train the one or moremachine learning models causes the one or more processors to pre-trainthe models before deployment and to perform training during deployment.

Example 12 is a method of analyzing test data, comprising: receiving oneor more user inputs through an interface identifying one or more test toperform on a device under test (DUT); forming a connection to at leastone test and measurement instrument; directing the test and measurementinstrument to perform one or more tests on the DUT and receive testresult data; applying one or more machine learning models to the testresult data to identify potentially anomalous test results; andgenerating and presenting a representation of the test result data andthe potentially anomalous test results.

Example 13 is the method of Example 12, further comprising storinginformation related to at least one of the test result data, machinelearning model configurations, metadata, and system logs.

Example 14 is the method of either of Examples 12 or 13, wherein forminga connection to at least one test and measurement instrument comprisesconnecting to an external test and measurement instrument using a driverfor the external test and measurement instrument.

Example 15 is the method of any of Examples 12 through 14, whereingenerating and presenting a representation comprises one of presenting agraph on a user interface, or presenting a data readout showinganomalous test results.

Example 16 is the method of any of Examples 12 through 15, whereinapplying one or more machine learning models comprises: generating anN-dimensional vector of the test result data for every test performed;and assigning an anomaly score to each data point in the test resultdata.

Example 17 is the method of any of Examples 12 through 16, whereinreceiving one more user inputs comprises receiving an intensity level.

Example 18 is the method of Example 17, wherein applying one or machinelearning models to identify anomalous test comprises identifyingoutliers depending upon the intensity level.

Example 19 is the Method of any of Examples 11 Through 18, FurtherComprising Training the One or More Machine Learning Models UsingPre-Trained Data Sets Prior to Model Deployment, and Data Sets DevelopedDuring Deployment of the Models

Example 20 is the method of Example 19, wherein training the one or moremachine learning models comprises uses overlapping data sets havingpre-defined window lengths of previous executed test results.

All features disclosed in the specification, including the claims,abstract, and drawings, and all the steps in any method or processdisclosed, may be combined in any combination, except combinations whereat least some of such features and/or steps are mutually exclusive. Eachfeature disclosed in the specification, including the claims, abstract,and drawings, can be replaced by alternative features serving the same,equivalent, or similar purpose, unless expressly stated otherwise.

Although specific embodiments have been illustrated and described forpurposes of illustration, it will be understood that variousmodifications may be made without departing from the spirit and scope ofthe disclosure. Accordingly, the invention should not be limited exceptas by the appended claims.

We claim:
 1. A test and measurement device, comprising: an interface;one or more connectors, each connector to allow the test and measurementdevice to connect to a test and measurement instrument; and one or moreprocessors, the one or more processors configured to execute code tocause the one or more processors to: receive one or more user inputsthrough the interface identifying one or more tests to perform on adevice under test (DUT); form a connection through one of the one ormore connectors to the DUT to perform the one or more tests and receivetest result data; apply one or more machine learning models to the testresult data to identify potentially anomalous test results; and generateand present a representation of the test result data and the potentiallyanomalous test results.
 2. The test and measurement device as claimed inclaim 1, further comprising one or more repositories to storeinformation related to at least one of the test result data, machinelearning model configurations, metadata, and system logs.
 3. The testand measurement device as claimed in claim 1, wherein the one or moreprocessors reside on a computing device separate from the test andmeasurement instrument.
 4. The test and measurement device as claimed inclaim 1, wherein the test and measurement device and the test andmeasurement instrument are the same device.
 5. The test and measurementdevice as claimed in claim 1, wherein the connectors comprise instrumentdrivers for one or more test and measurement instruments.
 6. The testand measurement device as claimed in claim 1, wherein the code thatcauses the one or more processors to apply one or more machine learningmodels comprises code that causes the one or more processors to:generate an N-dimensional vector of the test result data for every testperformed; and assign an anomaly score to each data point in the testresult data.
 7. The test and measurement device as claimed in claim 1,wherein the code the causes the one or more processors to receive theone or more user inputs comprises code that causes the one or moreprocessors to receive an intensity level.
 8. The test and measurementdevice as claimed in claim 7, wherein the code that cause the one ormore processors to apply one or more machine learning models to the testresult data to identify potentially anomalous test results causes theone or more processors to identify outliers depending upon the intensitylevel.
 9. The test and measurement device as claimed in claim 1, whereinthe one or more processors are further configured to execute code thatcauses the one or more processors to train the one or more machinelearning models.
 10. The test and measurement device as claimed in claim9, wherein the code that causes the one or more processors to train theone or more machine learning models uses multiple windows of apre-defined window length of previous executed test results.
 11. Thetest and measurement device as claimed in claim 9, wherein the code thatcauses the one or more processors to train the one or more machinelearning models causes the one or more processors to pre-train themodels before deployment and to perform training during deployment. 12.A method of analyzing test data, comprising: receiving one or more userinputs through an interface identifying one or more test to perform on adevice under test (DUT); forming a connection to at least one test andmeasurement instrument; directing the test and measurement instrument toperform one or more tests on the DUT and receive test result data;applying one or more machine learning models to the test result data toidentify potentially anomalous test results; and generating andpresenting a representation of the test result data and the potentiallyanomalous test results.
 13. The method as claimed in claim 12, furthercomprising storing information related to at least one of the testresult data, machine learning model configurations, metadata, and systemlogs.
 14. The method as claimed in claim 12, wherein forming aconnection to at least one test and measurement instrument comprisesconnecting to an external test and measurement instrument using a driverfor the external test and measurement instrument.
 15. The method asclaimed in claim 12, wherein generating and presenting a representationcomprises one of presenting a graph on a user interface, or presenting adata readout showing anomalous test results.
 16. The method as claimedin claim 12, wherein applying one or more machine learning modelscomprises: generating an N-dimensional vector of the test result datafor every test performed; assigning an anomaly score to each data pointin the test result data.
 17. The method as claimed in claim 12, whereinreceiving one more user inputs comprises receiving an intensity level.18. The method as claimed in claim 17, wherein applying one or machinelearning models to identify anomalous test comprises identifyingoutliers depending upon the intensity level.
 19. The method as claimedin claim 12, further comprising training the one or more machinelearning models using pre-trained data sets prior to model deployment,and data sets developed during deployment of the models
 20. The methodas claimed in claim 19, wherein training the one or more machinelearning models comprises uses overlapping data sets having pre-definedwindow lengths of previous executed test results.